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Abstract 

This study undertook the development and piloting of a coding system for the evaluation of 
asynchronous web-based instruction and learning. Processes were guided by four prominent 
educational perspectives and the extended examination of, and survey data from, seventeen 
archived web-based courses. These served as the bases for the development and application of the 
coding system. Data from the pilot application of the coding system indicate that there are 
particular features of courses favored by learners. Features such as content richness, instructor’s 
constructive and probing questions and responses, amount and quality of learner participation in 
discussions, and links to students’ own experiences are integral to those courses learners deem 
supportive of their learning and achievement. 
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Introduction 



Learning and teaching via the World Wide Web have become a convenient alternative to both 
classroom-based learning and traditional forms of distance education. However, the delivery of entire 
courses over the Internet is fairly new and consequently there are as yet few research tools available to 
use in analyzing the multiple dimensions of an online course to inform designers, instructors, researchers, 
and administrators. Findings from earlier quantitative analyses [1-2] of web-based courses indicate that 
students’ perceived learning is closely related to instructional design features such as the percentage of 
grade and specific requirements for participation in course discussions. Those studies, though offering 
valuable information about specific design features of online courses, do not provide details such as 1) 
the ways instructors design their questions for discussion; 2) how learners respond to scaffolds in 
discussions; and 3) how students interact with one another in discussion. To examine web-based courses 
at this level of detail, we developed and piloted an online discourse coding system. We employed 
qualitative and naturalistic methods in the hopes of gaining understandings about the nature of online 
learning that could not otherwise be gained through controlled experimentation [3], 

Design, development, and application of the coding system was the result of an attempt at capturing a 
deeper and broader view of the operations of web-based online courses. Data consisted of electronic 
transcripts of seventeen asynchronous web-based courses offered by the online learning network of a 
large eastern university system in 1 997 [4]. This paper presents an overview of the coding system for 
online courses, the process and rationale for its development, and the results of piloting its application. 

Perspective 

The guiding framework for the project was constructed from the following theoretical perspectives: 

• Sociocognitive views of learning 
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Based on the work of Soviet psychologist, Lev Vygotsky [5], this perspective sees learning as a socially 
mediated process. In an online environment, social interaction means the (asynchronous or synchronous) 
participation of all class members in the social construction of meaning [6], 

• A balance of BIG instruction and WIG instruction 

A BIG [7], or “beyond information given” approach involves some direct instruction of 
concepts; a WIG [7], or “without information given” approach withholds direct instruction of 
concepts and no correct answers are given to students. Jonassen [8] contends that instructional 
technologies can find an eclectic midpoint on the BIG-WIG continuum whereby to create 
appropriate environments for learning at particular levels. At an introductory level, for 
example, BIG can dominate; at a more advanced level, WIG can be the more dominant 
approach. At the level of mastery and expertise, WIG would predominate. 

• Cognitive flexibility 

Proposed by Spiro et al. [9], this view argues that since we cannot anticipate in advance the ways 
knowledge will be used in real life, our emphasis must be shifted from the retrieval of intact 
knowledge structures to the support for the construction of new understandings appropriate for 
specific and unique situations. Therefore, instruction should provide the learner with the 
opportunity of “revisiting the same material, at different times, in rearranged contexts, for different 
purposes, and from different conceptual perspectives.” [9, p. 66] Re-examining a case, for 
example, in a different context, will lead to new insights, new understandings, and new findings. 
This is because “partially nonoverlapping aspects of the case are highlighted” in different contexts. 

• Cognitive apprenticeship 

Cognitive apprenticeship [10] aims at making cognitive processes visible through teacher- 
student and student-student interaction. A collaborative online environment permits students to 
display and share their cognitive processes with onsite coaching, scaffolding, or modeling from 
the instructor. An online instructor is in a position to either provide coaching and scaffolding 
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to students on a one-on-one basis or in a public fashion. In an online course, it is possible for 
students and instructor to “meet” in a cognitive apprenticeship fashion several times a week. 

For our purposes, these four perspectives - sociocognitive views of learning, the BIG- WIG 
continuum, cognitive flexibility, and cognitive apprenticeship - do not stand alone as separate and 
unrelated pedagogies; rather, they can be seen as potentially complementary strategies through which one 
can approach the design and assessment of a learning environment relevant to its discourses. 

Methodology 

The goal of this project was to develop and apply a system of coding electronic transcripts archived 
from an initial nineteen undergraduate courses offered entirely online. The purposes for such an 
extensive undertaking (about thirty thousand actual documents) were multiple. We set out to develop a 
system that would 1) provide a systematic way of understanding the dynamics of online interaction in 
web-based courses; 2) provide feedback to designers and instructors regarding the effectiveness of their 
course design; and 3) demonstrate the potential and practicality of employing discourse coding strategies 
in evaluating this genre of teaching and learning. 

Currently there is an absence of comprehensive qualitative methodologies that can be applied to those 
digital documents generated in online courses. The existing coding schemes are either for classroom or 
computer conferencing only [1 1-17]. Therefore, in designing a tool to fill this need, we employed a 
tapestry approach, combining existing methodologies to develop a coding system for examining online 
discourse. Four broad categories of communication were identified that begin to capture the multiple 
dimensions of asynchronous online courses, namely: 

• course environment 

• instructor’s questions 

• instructor’s responses 



students’ responses. 



Course Environment 
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Categories of features unique to an online course environment were modeled after Hertz-Lazarowitz’s 
six interrelated classroom “mirrors” [12]: 1) physical organization of the classroom; 2) the learning task; 
3) teacher’s instructional behavior; 4) teacher’s communicative behavior; 5) students’ academic 
behavior; 6) students’ social behavior. The identification of six mirrors provides insights into the 
multiple dimensions of a course. However, the online course data for this study consist of electronic 
documents, rather than the physical organization of a classroom; therefore, a more specific coding system 
is needed, one that can code documents to see patterns of students’ online behaviors as well as 
instructor’s online acts. 

Instructors ’ Questions 

Using frameworks for sociocognitive and sociocollaborative learning theories, we examined 
instructors’ questions to see if they involve discourse practices that value open-ended and situated 
learning, authentic learning tasks, and students’ prior experiences. 

Instructors ’ Responses 

Categories of instructors’ responses identified by Ober et al. [1 1, see 18] were useful in systematically 
characterizing instructors’ responses. These features focus on the purpose of instructors’ responses - 
e.g., whether the response expands on information, invites further explanation, terminates the discussion, 
or manages other learning activities. 

Students ’ Responses 

In coding students’ responses, we adapted methodologies developed by Mason [13-16], 

Mason [13] made an attempt to draw up a typology of conference messages related to the educational 
values they display. This method involves a thorough reading of conferencing messages in order to 
discover what skills and abilities the participants are displaying or developing. Some of the questions 
suggested by Mason are very useful for studying students’ responses: 

- - do the participants build on previous messages? 

- - do they draw on their own experience? 
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- - do they refer to course material? 

- - do they refer to relevant material outside the course? 

- - do they initiate new ideas for discussion? 

[13, p.114] 

There have been further attempts at in-depth content analysis [14-15] of messages generated in 
computer conferencing. Henri [14] identified five dimensions of Computer-Mediated-Communication: 

1) the participative dimension; 2) the social dimension; 3) the interactive dimension; 4) the cognitive 
dimension; 5) the metacognitive dimension. Except for the social dimension, which is a separate section 
in the courses for this study and is not in the area of concern here, the dimensions identified by Henri are 
useful in examining students’ responses. Henri’s interactive dimension expects to show patterns in 
students’ online discussion; her cognitive and metacognitive dimensions are the focus of education in a 
socio-cognitive or constructivist point of view. 

Newman et al.’s study [15] elaborated the cognitive dimension of Henri’s and developed a coding 

system of critical indicators which aims at measuring the amount and type of critical thinking taking 

place in group learning in order to assess the possibilities of using computer-supported group learning 

(CSGL) to promote deep learning even in large classes. They developed a set of indicators of critical (+) 

or uncritical (-) thinking: 

R+- Relevance 
I+- Importance 

N+- Novelty. New information, ideas, solutions 

0+- Bringing outside knowledge/experience to bear on problem 

A+- Ambiguities: clarified or confused 

L+- Linking ideas, interpretation 

J+- Justification 

C+- Critical assessment 

P+- Practical utility 

W+- Width of understanding 

[15] 

To study the characteristics of the interactions in public conferences, Tsui and Ki [16] developed a 

framework of message analysis. They adopted the concept of turn from conversational analysis [19] as 

unit of analysis for conference messages and used speech act to identify message types. According to 

speech act theory, a speaker performs an action when he/she speaks [20-21 ]. In a conversational turn, a 
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speaker can perform different speech acts, such as asking a question, answering a question, making a 

request, and so forth. Similarly, in each message, a sender can perform such speech acts as asking a 

question, answering a question, or acknowledging a piece of information provided. Based on the 

examination of two conferences, Grammar Corner and Teaching Ideas Corner , Tsui and Ki classified 

conference messages into four categories 1) Questioning (Q); 2) Sharing (S); 3) Acknowledging (ACK); 

4) Others (O). They further subcategorized Questioning and Sharing into six sub-categories each: 

Questioning: 

asking about grammar 
asking about grammar teaching 
asking about teaching 
asking for comment 
asking for materials 
asking for information 
Sharing 

sharing views about grammar 
sharing views about teaching grammar 
sharing views about teaching 
giving comment 
sharing materials 
giving information 



Summary 

Each of the above mentioned methods has its own value and application in certain research contexts, 
though none was sufficient for this study. The scope of our study covers the various dimensions of an 
online course that differ from traditional classroom teaching and learning. Thus, we needed a special 
coding system for the analysis. We integrated and combined sources from these studies and developed a 
coding system that enabled us to easily and efficiently code course documents, instructor’s questions and 
responses, and students’ responses in terms of structure, speech acts, and content. 



The Coding System 

The development of the coding system was an inductive and iterative process; it involved not only 

extensive and intensive reading of the literature, but also repeated reading and comparison of the actual 
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course documents. Using online discussion activity as the focal point, we began with the four broad 
groupings: course environment, instructors’ questions, instructors’ responses, and students’ responses. 
We began with classifying all the course documents, coding them according to the first set of categories, 
using the inductive category coding strategy described by Maykut and Morehouse [22]. We 
experimented with coding all course documents for one course, then coded a second course, a third 
course, and so on. During the process, we discussed and revised the categories by adding, deleting, or 
modifying in order to best classify each of the course documents. We went through all the nineteen 
courses and then created a table for the revised set of categories. The same procedure was followed in 
developing categories for instructors” responses and instructors’ questions. For students’ responses, we 
used the same procedure but focused only on five selected courses [see 4] believing that this large 
volume of responses would be sufficiently representative of responses in other courses. 

Piloting the coding system 

In order to pilot the viability and potential utility of the coding system, we formulated the following 
research questions. Each corresponds to one of the four broad sets of coding categories: 

1 . What features of online course environments might contribute to students’ perceived 
learning online? [course environment] 

2. What types of instructor responses support online discourse? [instructors’ responses] 

3. What types of discussion questions elicit intensive discourse? [instructors’ questions] 

4. What are ideal students’ communication behaviors in online discourse? 

[students’ responses] 

Data collection 

Data for the pilot consisted of the archived transcripts of seventeen web-based courses delivered in 

the spring of 1997 (two having been eliminated from the original nineteen). Owing to the unruly volume 

of documents, we limited close analysis using the coding system to four of the seventeen courses. Criteria 
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for selecting these four courses were based in large part on learner survey responses pertaining to 
perceived learning (see Appendix A: Survey question for the selection of courses). Courses were 
arranged in descending order according to the class means for this item. Three courses from the top were 
selected (see Table 1). To contrast the salient features of these courses, one course from the bottom of 
the list was also included. This sampling method aims towards increased understanding versus 
generalizability of results [23, see 22], Considering possible differences in teaching strategies and the 
extent of discourse, the courses selected are close in subject matter: three social science courses and one 
business course. All are undergraduate courses with one at the underclass and three at the upperclass 
level. We designated the top three courses as High-perception courses and the bottom course as Low- 
perception course (see Table 1). 



Table 1 . Selected courses for qualitative analysis 





Upper-level 


Lower-level 


High-perception courses 

(From the top of the list of class means) 


Courses B, C 


Course A 


Low-perception course 

(From the bottom of the list of class means) 


Course D 





Analysis 

Once the coding system had been developed, a table was created for each set of categories with 
criteria for classifying documents or responses into that particular category [see 4]. We printed out the 
tables and started coding the archived files. We opened each document, read it, and put a mark in the 
category it was identified as belonging to. After we finished with one course, we tallied the marks in 
each cell. The same method was used for all four courses; we kept going back to the previous course or 
document to check consistency. Throughout the process, we continued to modify the categories and, if 
we found an extra one, we either created a new category or, in rare cases, put it in the category “Other.” 
After we finished the fourth course, we came back and read the documents in the four courses again to 
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check if the coding was accurate and revised as needed. In this way we established some reliability and 
consistency of coding across the four courses. 

One limitation related to the pilot is that we did not track nor analyze inter-rater reliability owing to 
the developmental status of the coding system. Additionally, the focus of the study was limited to testing, 
refining, and determining the utility of a coding system. A second limitation of the study is that, due to 
small sample size, we did not run any statistical analysis of the pilot data. Data presented are descriptive 
to provide a general picture of the features of the four courses. 

Results 

Course Environment 

— Insert Table 2 — 

Close examination of Table 2 shows that high-perception courses contain more content presentations. 

All courses except Course D created documents as content presentations, 59 documents in Course A 
as Additional readings or Commentary notes. Eleven in Course B as commentary notes in Reading 
assignments and 8 in Course C as Overview and Annotated Readings. In contrast, Course D did not have 
any documents that contained content presentation. 

Returning to our original framework, Courses A, B, C seemed to fall in the category of courses 
‘beyond the information given 5 (BIG) [7]; while Course D, a course without information given (WIG) 

[7]. This seems to indicate that some content support from the instructor might be necessary to make 
learning a positive experience at the undergraduate level. 

A second feature is that instructors in high-perception courses were more involved in discussion. 

Instructor A responded most often of all the four instructors, 59 times in total. Instructor B made 40 
responses, Instructor C 34 responses, and Instructor D 13. This aligns with Vygotskian views of the 
essentiality of the ‘more capable peer 5 and the support of more expert others through talk and activity 
form the nexus of learning. In the development of a cognitive skill which a learner has only partially 
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mastered, s/he can best employ it and eventually internalize it with the assistance and supervision of an 
expert [24, p.97], Moreover, the metaphor of apprenticeship stresses not only the learner's active role in 
learning, but also emphasizes the importance of a mentor's active support in learning a skill [10], 

A third emerging feature from these raw data is that high-perception courses placed more instructional 
emphasis on online discussion than did low-perception courses. Courses A, B, and C assigned 50%, 20%, 
and 50% of course grade weight to students’ participation in discussion. Although Course B’s percent 
was equal to Course D’s, it gave 5-point requirements for the quantity and quality of students’ 
participation in it. In addition, Course B required students to keep a weekly journal to reflect on a 
week’s reading and discussion. Course C merged the discussion and written assignments, using the 
discussion area as a public forum for students to discuss their research project proposals, findings, and 
final report. Otherwise solitary learning tasks were turned into interactive ones; the instructional 
emphasis was on students’ interaction with one another. 

Instructor ’s Questions 

— Insert Table 3 here — 

Three characteristics of instructor’s questions stand out from the data represented in Table 3: 

First, questions in high-perception courses sought a broad scope of answers and/or multiple 
perspectives on a topic. They were mostly open-ended or concerned controversial issues: eighteen out of 
twenty-one questions, or 86%, in Course A were open-ended and all the questions in Course B and C fell 
into this category. Seven out of the eighteen open-ended questions in Course A and eight out of ten in 
Course B required students to take a position and actively defend it in detail. 

Second, the structures of the questions in the high-perception courses were carefully and elaborately 
designed providing jumping off points for discussion, while questions in Course D were either single 
answer questions or textbook exercises. The branching method breaks down a question into sub- 
questions asking for information on the how, what, and why of a general problem; it opens students’ 
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minds to see multiple dimensions of an issue and provides substance for them to reflect on, ponder, 
discuss, and debate. 

Third, questions in the high-perception courses focused more on the students and the world around 
them. They were structured in a way that required students to reflect on and examine their inner 
thoughts, prior experience, or the world around them. Eighteen out of the 21 questions in Course A used 
the question structures What do you...? and/or How would you...? to probe students’ inner thoughts. 

Eight out of the ten questions in Course B and all the questions in Course C fell into this category. In 
each of the question documents, Instructor B stressed that students “think logically about the question” 
and respond by adding their “own thoughts and beliefs to the information presented in the book” [4], In 
these three courses, students are required to cite examples from their own experiences or real life 
situations to support their discussion. As each student brought in a slice of real life, they were 
collectively building a database of situations to which the constructs in the discussion could apply. 

Course D had a much simpler question structure. Although one third of the questions in Course D sought 
students’ personal responses, none of the questions required students to examine their own experiences in 
order to answer the questions. The questions were mostly broad and general, detached from students’ 
prior experiences and real life contexts. 

Instructor ’s Responses 

-- Insert Table 4 here - 

Table 4 reveals that Course A, a lower-level course, had an overwhelmingly large number of 
responses in the categories of Expand thinking and Extend thinking, while Courses B and C, two 
upperclass courses had more responses at the management level. These two types of responses are 
designed to model and facilitate or scaffold discussion at the content level. Extensive use of Expand and 
Extend responses in Course A seemed to indicate that students at a lower academic level needed more 
input at the content level from the instructor to keep the discussion going. Responses in Course B and 



Course C were mostly at the Manage level. It seems that students in underclass courses had not had 
sufficient content knowledge to carry on a discussion without an instructor’s content scaffolding. 
Conversely, in an upperclass course, students are more developed in terms of content knowledge so that 
they may be able to carry on a discussion with less instructor input. However, the more advanced the 
level of a course, the more complex the discussion assignments might become. Students may thereby still 
need an instructor’s constant scaffolding at the Manage level. 

Students ’ Responses 

— Insert Table 5 here -- 

Data in Table 5 - students’ responses - indicate that a large number of responses in the high- 
perception courses were related to students’ own personal experience or examples familiar to students. 
Thirty-two percent of the responses in Course A fell into this category, 36% in Course B, and 20% in 
Course C belong to this category. Only 14% in Course D fell into this category. This was consistent 
with the results of the analysis on instructor’s questions. It was found that more questions in high- 
perception courses sought answers based on students’ prior experience. Here students were encouraged 
to bring in cases or contexts they had experienced or were familiar with and collaboratively build a 
database of situations to which the constructs discussed could be applied. Encountering the same 
materials “in a different context or from a different intentional point of view will enable the learner to 
grasp important knowledge facets that may have escaped notice at the first encounter. Reexamining a 
case in a different context will lead to new insights, new understandings, and new findings. This is 
because different aspects of the case are highlighted in different contexts” [9, p. 66], 

The categories of students’ responses in Table 5 were arranged in a hierarchical order from 
cognitively less demanding responses lik e Accept or Understand, to cognitively more demanding 
responses like Link or Plan. In Table 5, more than 77% of students’ responses in Course B, and more 
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than 80% in Course C were beyond the simple understanding level, while more than 60% in Course D 
were at the simple understanding level. 

The results of the patterns of students 5 responses seem to match those of the structure of instructor’s 
questions. As was found above, questions in the high-perception courses sought broader answers, related 
more to students’ prior experience and the real world around them, and had a more careful and elaborate 
design (branching, using issues, or choices, or simulation). These features might have influenced 
patterns of students’ responses. 

Since this qualitative analysis is based on four courses, it cannot purport any cause-effect relation 
between patterns of students’ responses and the structure of instructors’ questions. A combination of 
qualitative and quantitative analysis of a larger sample size is needed for a test of the relationship 
between them. For the purpose of this study, it suffices to highlight those salient features of the high- 
perception courses as a basis for further inquiry. 

Discussion and conclusions 

This pilot application of a coding system designed for analyzing effective web-based course features 
and dynamics enabled us to gain insights into effective instructional design features and the 
interdependent nature of instructors’ and students’ online communication behaviors. Through carefully 
analyzing the anatomy of courses using the four broad sets of categories, it appears that students were 
more satisfied with a course environment in which they were provided content and opportunity to 
construct meanings through social interaction. However, the degree to which a course was collaborative 
was limited by the academic level of students. Students at the lower level appeared to need more 
instructor scaffolding than those at higher academic levels. In addition, since all four courses were at the 
introductory level, a certain amount of content input seemed necessary to ensure and support students’ 
active learning, though the amount varied according to the academic level of a course. An entirely 
“empty” technology [25] or WIG [7] course environment might negatively influence students’ perception 




14 



15 



of their learning experiences. A course with a balance between “full” [25] and “empty” technologies or 
between BIG and WIG [7] with emphasis on students’ social interaction seems more appropriate at the 
introductory level. 

Patterns of students’ responses correspond to the structure and nature of instructor’s discussion 
questions. The manner in which discussion questions were structured was reflected in the ways students 
responded, and ultimately may have been reflected in how students perceived their learning experiences 
in the courses. Probing students’ inner thoughts, relating to students’ prior experience, using 
controversial issues, simulation, or research projects as discussion topics might have elicited responses 
that went beyond mere understanding of textbook materials. Textbook exercises or simple discussion 
questions that were detached from real world or students’ prior experiences might have limited and 
restricted students’ critical thinking and, as a result, kept the discussion at a low level in terms of 
cognitive demand. 

Since the objective of the project is to develop a coding system, the pilot analysis has its limitations. 
Small sample size and lack of inter-rater reliability for the coding process require that the findings from 
the piloting should be taken with caution. The study has achieved its goal if our pilot findings serve as a 
catalyst for further inquiries and provide a framework for online course coding systems to be expanded 
and refined. 
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Appendix A: Survey question for the selection of courses 



The question from the survey for the selection of the four courses: 



Compared to other modes of learning, do you think you have learned more in this course? 

A. considerably more in this course 

B. more in this course 

C. about the same amount in this course 

D. less in this course 

E. much less in this course 
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Table 2. Characteriatics of course environments (Numbers in the table are all raw counts,) 









High-perception 




Low- 












perception 


Categories 


Description 


Course A 


Course B 


Course C 


Course D 






(Lower)* 


(Upper)* 


(Upper)* 


(Upper)* 


Number of students 




6 


23 


11 


10 


1. Instructor 












contributions 












Content 

presentations 


This includes mini-lectures and presentation inside reading 
assignments documents: total number and average length 
of each/ use computer screen as unit. 


59 (Doc) 
-174 
(screens) 


1 1 (Doc) = 
75 (screens) 


8 (Doc) = 27 
(screens) 


0 


Evaluation 


This refers to instructor evaluation of students’ written 
assignments or other learning activities 


46 


192 


42 


42 


Responses 


Responses in discussion area 


59 


40 


34 


13 


2. Design of learning 
activities 


Written 

assignments 


Grade percent assigned to written assignments 


50% 


80% 


50% 


80% 


Types 


essay 




1 








paper 






2 


3 


case study 1 


journal ] | 




quiz/exam (multiple choice = MC; question answer « QA; 
true-false = TF; problem solving = PS) 


7 (MC, QA) 


3 (QA) 








self-assessment quiz (multiple choice = MC; question 
answer = QA) 






30 (QA) 






other 






l(memo) 


3(Web- 

research) 


Nature 


non-interactive: submitted to professor forevaluation only 
and invisible to other students 


7 


14 


32 


3 




interactive: submitted to class for comments or discussion 




1 (Self- 


2 






and visible to all members in the class including the 
instructor. 




Introduction) 






interactive: group projects. Students work on the 
assignments in groups and present them to the class or the 
professor as a group 


Online 

discussion 


grade percent assigned to participation in discussion 


50% 


20% 


50% 


20% 




quality and quantity requirements: Six levels of 


5 


5 


4 


3 



instructor's requirements of students’ participation in 
discussion are identified from the courses: with a value 
from 0 as the lowest and 5 as the highest. 



3. Students’ responses 


number of contributions by students per module: a raw 


**32; 31; 


**164; 124; 


**56; 110; 


**45; 90; 


in discussion 


count of responses in each module 


49; 46; 32; 
36; 39 


162; 110; 
108 


55; 149 


71 ; 44; 55 



* indicates the academic level of a course: "Lower” means an undergraduate course at the lower level; "Upper” means an undergraduate 
course at the upper level. 

** each number indicates the number of responses in a module. 



BEST COPY AVAILABLE 
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Table 3. Instructor's questions (raw counts) 









High-perception 




Low- 












perception 


Categories 


Definitions 


Course A 


Course B 


Course C 


Course D 






(Lower)* 


(Upper)* 


(Upper) 


(Upper) 


Scope of answers 
sought 


Closed: use simple question structures like Do you...? Is .... which can 
well be answered using "yes” or "no.” or use “What." “Who,” "List” 
questions that seek an exhaustive list of things in the answers. (Close) 


2 






6 




Open-ended ideas 

using: How...? Why? or using the word “explain,” “describe,” 
“comment ” (Open) 


18 


10 


6 


9 




**Issues 

asking students to take stand between opposite or 
differing views, or to compare opposite views (OppVw) 


7 


8 


l 


5 


Structure 


single focused question (Single) 


6 


0 


0 


It 




branching questions on one topic (Branching) 


15 




6 


2 




multiple topics for students to select to respond (Multiple) 




10 








not questions, but a description of the discussion assignment (NoQu) 






l 


2 


requiring role play or small group discussion (RolePlay) 1 


Relation to written 
assignments 


using written assignments as discussion topics (UsWAssign) 






3 






relating to, or related with, written assignments (TWritt, WWritt) 




***10 

WWritt 






Relation to students 


probing students' personal thoughts or experiences: using What do you 
think...? How would you...? Based on your experience.., or From your 
own experience... (ProbStu) 


20 


8 


6 


5 


Relation to real life 


requiring examples from real life to support arguments (ReExample) 




8 








relating to real life situations, providing a scenario for discussion 
(RealSitu) 


l 


3 


2 





* indicates the academic level of a course: "Lower” means an undergraduate course at the lower level; “Upper” means an undergraduate 
course at the upper level. 

** indicates a sub-category of the above category 

*** indicates that ten written assignments referred students to ten discussion topics. 

**** indicates that six discussion questions referred students to six written assignments. 



BEST COPY AVAILABLE 
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Table 4. Instructor’s responses (raw counts) 







High-perception 




Low- 

perception 


Categories 


Description 


Course A Course B 


Course C 


Course D 






(Lower)* (Upper)* 


(Upper) 


(Upper) 


Instruct 


The instructor responds to correct a student’s opinion or viewpoint 
as an authoritative person, to express an authoritative idea to give 
an authoritative explanation, or to cite an authority; or to correct 
students’ understanding by indicating that students should think in 
a certain way (Instruct). 








Critically evaluate 
thinking 


The instructor responds to critically evaluate students’ thinking; 
You are being too critical ..; you may be thinking narrowly of... 
(Eval/N). 








Positively evaluate 
thinking 


The instructor responds to positively evaluate students’ thinking: 
You made a strong point...; 1 agree with you ... (Eval/P). 


2 


1 




Expand thinking 


The instructor responds to add information on a subject, to clarify, 
build, or develop ideas suggested by students, or answer content 
questions (Expand). 


31 9 


1 


1 


Extend thinking 


The instructor responds to extend level of discussion by asking 
questions or requesting further information (Extend). 


24 2 


3 


3 


Summarize 


The instructor responds to summarize the thread of discussion. 








Maintain participation 


The instructor responds to a student to maintain level of discussion 
by inviting students to continue discussion on a topic (Maint). 


1 






Holistically Praise or 
encourage 


The instructor responds to holistically praise or encourage students, 
such as Good job! You are on the right track! Keep up the good 
work! (Praise) 


1 3 


6 


1 


Manage learning 
activities 


The instructor responds to manage learning activities - comments 
that organize learning activities, give assignments or instructions 
on how to do the assignments (Manage). 


1 29 


24 


5 


Other 


The instructor responds to talk about something that is not related 
to the discussion topic or any learning activities. 


1 


1 


1 


♦ indicates the academic level of a course: "Lower” means an undergraduate course at the lower level; “Upper” means an undergraduate 
course at the upper level. 



BEST COPY AVAILABLE 
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Table 5. Students’ responses (raw counts) 







High-perception 




Low-perception 


Students’ responses 


Course A 
N= 95 


Course B 
N = 286 


Course C 
N = 259 


Course D 
N = 161 


Respond to talk about non-content related subject 
(NONContent) 


2 


26=9% 


14=5% 




Respond to accept views and opinions, or make 
simple value judgments without adding further 
comments (Accept) 




13 


22=8% 


3 


Respond to discuss personal understanding of the 
question or problem, adding information 
(Understanding) 


*33=34% 


*38=13% 


8 


*97=60% 


Respond to bring personal experiences or outside 
examples to bear on problems (Experience) 


*30=32% 


*102=36% 


*53=20% 


23=14% 


Respond to accept views, opinions, ideas and add 
further comments or personal interpretations 
(Comment) 


8=8% 


*57=20% 


*44=17% 


17=11% 


Respond to ask questions to initiate new discussion 
or elicit feedback and comments (Question) 


8=8% 


15=5% 


*35=14% 


4 


Respond to express different views, opinions, 
interpretations on problems (Differ) 


6 


7 


7 


11=7% 


Respond to answer students’ questions, justify 
positions, discuss ambiguity or 
misunderstanding to clear them up (Clear) 


6 


12 


20=8% 


2 


Respond to analyze students’ views, opinions, and 
make suggestions (Analyze) 


4 


1 5=5% 


29=1 1% 


5 


Respond to link facts, ideas, and notions to make 
inference; to synthesize ideas or discussion (Link) 




5 


*24=9% 




Respond to discuss plans or procedures for working 
on a task (Plan) 






11=4% 





* indicates categories that have high percentages and make up 60% of responses in a course 
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